Notebook metadata changed
xxxxxxxxxx
29
 
1
​
15
    "nbconvert_exporter": "python",
16
    "pygments_lexer": "ipython3",
17
    "version": "3.8.1"
18
  },
19
  "pycharm": {
20
​
⇛⇚
xxxxxxxxxx
29
 
1
​
15
    "nbconvert_exporter": "python",
16
    "pygments_lexer": "ipython3",
17
    "version": "3.8.3"
18
  },
19
  "pycharm": {
20
​

Installation¶

Just pip install:

pip install omegaconf

If you want to try this notebook after checking out the repository be sure to run python setup.py develop at the repository root before running this code.

Creating OmegaConf objects¶

Empty¶

In [1]:
xxxxxxxxxx
3
 
1
from omegaconf import OmegaConf
2
conf = OmegaConf.create()
3
print(conf)
Outputs unchanged
{}
application/vnd.jupyter.stdout

From a dictionary¶

In [2]:
xxxxxxxxxx
2
 
1
conf = OmegaConf.create(dict(k='v',list=[1,dict(a='1',b='2')]))
2
print(OmegaConf.to_yaml(conf))
Outputs unchanged
k: v
list:
- 1
- a: '1'
  b: '2'
application/vnd.jupyter.stdout

From a list¶

In [3]:
xxxxxxxxxx
2
 
1
conf = OmegaConf.create([1, dict(a=10, b=dict(a=10))])
2
print(OmegaConf.to_yaml(conf))
Outputs unchanged
- 1
- a: 10
  b:
    a: 10
application/vnd.jupyter.stdout

From a yaml file¶

In [4]:
xxxxxxxxxx
2
 
1
conf = OmegaConf.load('../source/example.yaml')
2
print(OmegaConf.to_yaml(conf))
Outputs unchanged
server:
  port: 80
log:
  file: ???
  rotation: 3600
users:
- user1
- user2
application/vnd.jupyter.stdout

From a yaml string¶

In [5]:
xxxxxxxxxx
9
 
1
yaml = """
2
a: b
3
b: c
4
list:
5
- item1
6
- item2
7
"""
8
conf = OmegaConf.create(yaml)
9
print(OmegaConf.to_yaml(conf))
Outputs unchanged
a: b
b: c
list:
- item1
- item2
application/vnd.jupyter.stdout

From a dot-list¶

In [6]:
xxxxxxxxxx
3
 
1
dot_list = ["a.aa.aaa=1", "a.aa.bbb=2", "a.bb.aaa=3", "a.bb.bbb=4"]
2
conf = OmegaConf.from_dotlist(dot_list)
3
print(OmegaConf.to_yaml(conf))
Outputs unchanged
a:
  aa:
    aaa: 1
    bbb: 2
  bb:
    aaa: 3
    bbb: 4
application/vnd.jupyter.stdout

From command line arguments¶

To parse the content of sys.arg:

In [7]:
xxxxxxxxxx
5
 
1
# Simulating command line arguments
2
import sys
3
sys.argv = ['your-program.py', 'server.port=82', 'log.file=log2.txt']
4
conf = OmegaConf.from_cli()
5
print(OmegaConf.to_yaml(conf))
Outputs unchanged
server:
  port: 82
log:
  file: log2.txt
application/vnd.jupyter.stdout

Access and manipulation¶

Input yaml file:

In [8]:
xxxxxxxxxx
2
 
1
conf = OmegaConf.load('../source/example.yaml')
2
print(OmegaConf.to_yaml(conf))
Outputs unchanged
server:
  port: 80
log:
  file: ???
  rotation: 3600
users:
- user1
- user2
application/vnd.jupyter.stdout

Object style access:¶

In [9]:
xxxxxxxxxx
1
 
1
conf.server.port
Outputs unchanged
80
text/plain

dictionary style access¶

In [10]:
xxxxxxxxxx
1
 
1
conf['log']['rotation']
Outputs unchanged
3600
text/plain

items in list¶

In [11]:
xxxxxxxxxx
1
 
1
conf.users[0]
Outputs unchanged
'user1'
text/plain

Changing existing keys¶

In [12]:
xxxxxxxxxx
1
 
1
conf.server.port = 81

Adding new keys¶

In [13]:
xxxxxxxxxx
1
 
1
conf.server.hostname = "localhost"

Adding a new dictionary¶

In [14]:
xxxxxxxxxx
1
 
1
conf.database = {'hostname': 'database01', 'port': 3306}

providing default values¶

In [15]:
xxxxxxxxxx
2
 
1
conf.missing_key or 'a default value'
2
conf.get('missing_key', 'a default value')
Outputs unchanged
'a default value'
text/plain

Accessing mandatory values¶

Accessing fields with the value ??? will cause a MissingMandatoryValue exception. Use this to indicate that the value must be set before accessing.

In [16]:
xxxxxxxxxx
5
 
1
import pytest
2
from omegaconf import MissingMandatoryValue
3
​
4
with pytest.raises(MissingMandatoryValue):
5
    conf.log.file
xxxxxxxxxx
5
 
1
# Variable interpolation
2
​
3
OmegaConf support variable interpolation, Interpolations are evaluated lazily on access.
4
​
5
### Config node interpolation
⇛⇚
xxxxxxxxxx
5
 
1
# Variable interpolation
2
​
3
OmegaConf support variable interpolation, Interpolations are evaluated lazily on access.
4
​
5
## Config node interpolation

Interpolations are evaluated lazily on field access.
Note below that when printed the interpolations are not resolved.
They get resolved once you access them.

In [17]:
xxxxxxxxxx
2
 
1
conf = OmegaConf.load('../source/config_interpolation.yaml')
2
print(OmegaConf.to_yaml(conf))
Outputs unchanged
server:
  host: localhost
  port: 80
client:
  url: http://${server.host}:${server.port}/
  server_port: ${server.port}
application/vnd.jupyter.stdout
In [18]:
xxxxxxxxxx
4
 
1
# Primitive interpolation types are inherited from the referenced value
2
print("conf.client.server_port: ", conf.client.server_port, type(conf.client.server_port).__name__)
3
# Composite interpolation types are always string
4
print("conf.client.url: ", conf.client.url, type(conf.client.url).__name__)
Outputs unchanged
conf.client.server_port:  80 int
conf.client.url:  http://localhost:80/ str
application/vnd.jupyter.stdout
xxxxxxxxxx
1
 
1
to_yaml will resolve interpolation if `resolve=True` is passed
⇛⇚
xxxxxxxxxx
1
 
1
`to_yaml()` will resolve interpolations if `resolve=True` is passed
In [19]:
xxxxxxxxxx
1
 
1
print(OmegaConf.to_yaml(conf, resolve=True))
Outputs unchanged
server:
  host: localhost
  port: 80
client:
  url: http://localhost:80/
  server_port: 80
application/vnd.jupyter.stdout
xxxxxxxxxx
1
 
1
Interpolations may be nested, enabling more advanced behavior like dynamically selecting a sub-config:
Metadata changed
xxxxxxxxxx
In [20]:
xxxxxxxxxx
10
 
1
cfg = OmegaConf.create("""
2
plans:
3
    A: plan A
4
    B: plan B
5
selected_plan: A
6
plan: ${plans.${selected_plan}}
7
""")
8
print(f"Default: cfg.plan = {cfg.plan}")
9
cfg.selected_plan = "B"
10
print(f"After selecting plan B: cfg.plan = {cfg.plan}")
Metadata changed
xxxxxxxxxx
Default: cfg.plan = plan A
After selecting plan B: cfg.plan = plan B
application/vnd.jupyter.stdout
xxxxxxxxxx
5
 
1
### Environment variable interpolation
2
​
3
Environment variable interpolation is also supported.
4
​
5
Input yaml file:
⇛⇚
xxxxxxxxxx
3
 
1
## Environment variable interpolation
2
​
3
Environment variable interpolation is also supported.
In [20]:
In [21]:
xxxxxxxxxx
3
 
1
# Let's set up the environment first (Only needed for this demonstration)
2
import os
3
os.environ['USER'] = 'omry'
⇛⇚
xxxxxxxxxx
3
 
1
# Let's set up the environment first (only needed for this demonstration)
2
import os
3
os.environ['USER'] = 'omry'

Here is an example config file interpolates with the USER environment variable:

In [21]:
In [22]:
xxxxxxxxxx
2
 
1
conf = OmegaConf.load('../source/env_interpolation.yaml')
2
print(OmegaConf.to_yaml(conf))
Outputs unchanged
user:
  name: ${env:USER}
  home: /home/${env:USER}
application/vnd.jupyter.stdout
In [22]:
In [23]:
xxxxxxxxxx
2
 
1
conf = OmegaConf.load('../source/env_interpolation.yaml')
2
print(OmegaConf.to_yaml(conf, resolve=True))
Outputs unchanged
user:
  name: omry
  home: /home/omry
application/vnd.jupyter.stdout
xxxxxxxxxx
1
 
1
You can specify a default value to use in case the environment variable is not defined. The following example sets `abc123` as the the default value when `DB_PASSWORD` is not defined.
Metadata changed
xxxxxxxxxx
In [24]:
xxxxxxxxxx
6
 
1
os.environ.pop('DB_PASSWORD', None)  # ensure env variable does not exist
2
cfg = OmegaConf.create({'database': {'password': '${env:DB_PASSWORD,abc123}'}})
3
print(repr(cfg.database.password))
4
OmegaConf.clear_cache(cfg) # clear resolver cache
5
os.environ["DB_PASSWORD"] = 'secret'
6
print(repr(cfg.database.password))
Metadata changed
xxxxxxxxxx
'abc123'
'secret'
application/vnd.jupyter.stdout
xxxxxxxxxx
1
 
1
Environment variables are parsed when they are recognized as valid quantities that may be evaluated (e.g., int, float, dict, list):
Metadata changed
xxxxxxxxxx
In [25]:
xxxxxxxxxx
10
 
1
cfg = OmegaConf.create({'database': {'password': '${env:DB_PASSWORD,abc123}',
2
                                    'user': 'someuser',
3
                                    'port': '${env:DB_PORT,3306}',
4
                                    'nodes': '${env:DB_NODES,[]}'}})
5
os.environ["DB_PORT"] = '3308'
6
print(repr(cfg.database.port))  # converted to int
7
os.environ["DB_NODES"] = '[host1, host2, host3]'
8
print(repr(cfg.database.nodes))  # converted to list
9
os.environ["DB_PASSWORD"] = 'a%#@~{}$*&^?/<'
10
print(repr(cfg.database.password))  # kept as a string
Metadata changed
xxxxxxxxxx
3308
['host1', 'host2', 'host3']
'a%#@~{}$*&^?/<'
application/vnd.jupyter.stdout
xxxxxxxxxx
1
 
1
You can also set a default value for environment variables:
Metadata changed
xxxxxxxxxx
In [23]:
xxxxxxxxxx
2
 
1
conf = OmegaConf.create({"user" : {"age" : "${env:AGE, 42}"}})
2
print(conf.user.age)
Metadata changed
xxxxxxxxxx
42
application/vnd.jupyter.stdout

Custom interpolations¶

xxxxxxxxxx
1
 
1
You can add additional interpolation types using custom resolvers. This example creates a resolver that adds 10 the the given value.
⇛⇚
xxxxxxxxxx
2
 
1
You can add additional interpolation types using custom resolvers.
2
The example below creates a resolver that adds 10 to the given value.
In [24]:
In [26]:
xxxxxxxxxx
3
 
1
OmegaConf.register_resolver("plus_10", lambda x: int(x) + 10)
2
conf = OmegaConf.create({'key': '${plus_10:990}'})
3
print(conf.key)
⇛⇚
xxxxxxxxxx
3
 
1
OmegaConf.new_register_resolver("plus_10", lambda x: x + 10)
2
conf = OmegaConf.create({'key': '${plus_10:990}'})
3
conf.key
Outputs changed
Output added
1000
text/plain
Output deleted
1000
application/vnd.jupyter.stdout
xxxxxxxxxx
1
 
1
You can take advantage of nested interpolations to perform custom operations over variables:
Metadata changed
xxxxxxxxxx
In [27]:
xxxxxxxxxx
3
 
1
OmegaConf.new_register_resolver("plus", lambda x, y: x + y)
2
conf = OmegaConf.create({"a": 1, "b": 2, "a_plus_b": "${plus:${a},${b}}"})
3
conf.a_plus_b
Metadata changed
xxxxxxxxxx
3
text/plain
xxxxxxxxxx
1
 
1
By default a custom resolver’s output is cached, so that when it is called with the same inputs we always return the same value. This behavior may be disabled by setting `use_cache=False`:
Metadata changed
xxxxxxxxxx
In [28]:
xxxxxxxxxx
18
 
1
import random
2
random.seed(1234)
3
​
4
OmegaConf.new_register_resolver("randint",
5
                                lambda a, b: random.randint(a, b))
6
c = OmegaConf.create({"x": "${randint:0, 1000}"})
7
print("With cache:")
8
print(f"c.x = {repr(c.x)}")
9
print(f"c.x = {repr(c.x)}")  # same as above thanks to the cache
10
​
11
random.seed(1234)
12
OmegaConf.new_register_resolver("randint_nocache",
13
                                lambda a, b: random.randint(a, b),
14
                                use_cache=False)  # <== disable cache!
15
c = OmegaConf.create({"x": "${randint_nocache:0, 1000}"})
16
print("Without cache:")
17
print(f"c.x = {repr(c.x)}")
18
print(f"c.x = {repr(c.x)}")  # not the same anymore since the cache is disabled
Metadata changed
xxxxxxxxxx
With cache:
c.x = 989
c.x = 989
Without cache:
c.x = 989
c.x = 796
application/vnd.jupyter.stdout

Merging configurations¶

Merging configurations enables the creation of reusable configuration files for each logical component instead of a single config file for each variation of your task.

Machine learning experiment example:

conf = OmegaConf.merge(base_cfg, model_cfg, optimizer_cfg, dataset_cfg)

Web server configuration example:

conf = OmegaConf.merge(server_cfg, plugin1_cfg, site1_cfg, site2_cfg)

The following example creates two configs from files, and one from the cli. It then combines them into a single object. Note how the port changes to 82, and how the users lists are combined.

In [25]:
In [29]:
xxxxxxxxxx
2
 
1
base_conf = OmegaConf.load('../source/example2.yaml')
2
print(OmegaConf.to_yaml(base_conf))
Outputs unchanged
server:
  port: 80
users:
- user1
- user2
application/vnd.jupyter.stdout
In [26]:
In [30]:
xxxxxxxxxx
2
 
1
second_conf = OmegaConf.load('../source/example3.yaml')
2
print(OmegaConf.to_yaml(second_conf))
Outputs unchanged
log:
  file: log.txt
application/vnd.jupyter.stdout
In [27]:
In [31]:
xxxxxxxxxx
11
 
1
from omegaconf import OmegaConf
2
import sys
3
​
4
# Merge configs:
5
conf = OmegaConf.merge(base_conf, second_conf)
6
​
7
# Simulate command line arguments
8
sys.argv = ['program.py', 'server.port=82']
9
# Merge with cli arguments
10
conf.merge_with_cli()
11
print(OmegaConf.to_yaml(conf))
Outputs unchanged
server:
  port: 82
users:
- user1
- user2
log:
  file: log.txt
application/vnd.jupyter.stdout
In [ ]:
xxxxxxxxxx
1
 
1
​
Metadata changed
xxxxxxxxxx